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ABSTRACT 

Robotics  has  developed  with  great  vigor  in  recent  years,  particularly  due  to  the  great  interest  of  researchers,  who  seek  a 
deeper  integration  of  these  artificial  systems  in  the  daily  life  of  human  beings.  One  of  the  areas  that  is  expected  to  have 
the  greatest  impact  on  society  is  service  robotics.  In  this  area,  robotic  platforms  are  developed  to  perform  tasks  that 
support  the  daily  activity  of  people,  for  example,  caring  for  children  and  elderly  people.  This  type  of  task  presents  very 
specific  design  challenges.  For  example,  the  robot  must  be  able  to  move  in  the  same  environment  as  the  human  user, 
however,  these  types  of  environments  turn  out  to  be  dynamic  and  unknown.  The  robot  must  somehow  securely  identify 
each  of  the  obstacles  in  the  environment  (at  least  those  close  by)  and  define  action  strategies  based  on  this  information. 
We  propose  a  motion  planning  scheme  for  an  anthropomorphic  robotic  platform  that  relies  on  the  visual  identification  of 
specific  elements  in  the  environment.  From  this  identification,  the  strategy  defines  real-time  movement  policies  that 
facilitate  the  programming  of  tasks  in  the  robot.  The  strategy  was  evaluated  in  the  laboratory  on  our  robotic  platform, 
demonstrating  high  performance  and  low  computational  cost. 
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INTRODUCTION 

Human  development  has  been  strongly  marked  by  its  sense  of  sight  [1],  This  is  the  most  important  sensor  for 
interacting  with  the  environment  and  through  it,  intelligence  has  been  developed,  and  today,  it  is  a  way  of 
transforming  the  environment  through  engineering  and  development.  This  importance  is  easily  observed  in 
everyday  life.  For  example,  when  a  person,  for  the  first  time,  is  in  someplace  (cinema,  hotel,  etc.),  his  first  reaction 
is  to  identify  visual  elements,  previously  known  to  trace  a  strategy  of  movement  in  this  new  environment  (look  for 
the  ticket  office  or  the  reception  office).  This  basic  principle  of  self-localization  can  be  implemented  under  the  same 
principles  on  autonomous  robots  [2], 

Great  advances  in  real-time  machine  vision  and  image  processing  schemes  have  strongly  marked  the 
development  of  autonomous  navigation  systems  for  robots.  These  developments  are  mainly  due  to  the  increased 
performance  of  embedded  systems  and  optical  sensors  with  their  consequent  reduction  in  cost.  At  the  same  time, 
new  image  processing  schemes  have  also  emerged,  including  deep  neural  networks  with  great  possibilities  of 
filtering  and  classification  in  real-time  [3],  These  tools  are  adapted  to  the  specific  needs  of  the  robots  and  desired 
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tasks.  For  example,  in  the  service  robotics,  tasks  are  expected  to  be  performed  in  indoor  human  environments  in  unfamiliar 
areas  for  the  robot  (but  observable  through  sensors,  optical  sensors,  for  example),  and  with  a  high  probability  of  continuous 
change  (dynamic  environments)  that  make  programming  or  prior  mapping  difficult  [4,  5].  These  characteristics  can  also  be 
found  in  other  tasks,  in  fact,  in  industrial  environments,  there  are  the  same  problems  for  the  path  planning  of  robots,  and 
just  as  for  service  robots,  the  solution  lies  in  the  adoption  of  reactive  strategies  from  local  readings  [6,  7].  Under  this 
scheme,  the  robot  defines  its  movement  based  on  the  information  collected  from  the  environment,  that  is,  it  decides  where 
to  move  and  how  to  do  it  if  it  identifies  some  important  element  for  the  development  of  its  task. 

These  robots  must  possess  certain  processing  and  control  characteristics  that  allow  them  to  move  in  the 
environment.  For  example,  the  robot  must  have  a  basic  function  of  exploring  the  environment  through  which  it  can  sense 
distinctive  elements  from  shapes,  colors,  sizes  or  any  other  noticeable  elements  (or  combination  of  them).  With  this 
information,  the  robot  can  carry  out  a  partial  (or  even  global)  reconstruction  of  the  environment  without  its  previous 
knowledge  [8].  This  process  is  usually  called  self-localization  and  corresponds  to  the  positioning  with  respect  to  some  local 
frame  of  reference,  the  landmark  identified  by  the  sensors  [9].  From  this  information,  the  robot  can  establish  distances, 
relationships  and  define  movement  strategies  in  coherence  with  its  task. 

Other  types  of  tasks  may  have  similar  characteristics  in  terms  of  the  problems  to  be  solved.  This  is  the  case  of 
Unmanned  Aerial  Vehicles  (UAVs),  which  independently  of  their  task  (surveillance,  photo  grammetry,  supervision,  etc.) 
or  control  scheme,  use  similar  localization  mechanisms.  These  aerial  robots  also  track  specific  elements  in  the  terrain 
below  them  to  find  landmarks  that  coordinate  their  flight  plans.  In  particular,  they  first  use  low-resolution  images  to  find 
these  specific  landmarks  and  then  optimize  the  reading  of  the  images  to  be  used  as  a  reference  in  the  design  of  the 
navigation  route  [10,  11]. 

It  is  also  possible  to  use  other  types  of  images,  beyond  those  generated  by  optical  transducers  based  on  light,  for 
example,  using  images  produced  by  ultrasound  systems.  These  images  can  also  be  used  to  process  and  define  navigation 
strategies  and  have  been  used  successfully  in  medicine  [12,  13].  Ultrasound  systems  are  capable  of  producing  three- 
dimensional  images  (containing  3D  tissue  information)  with  high  depth  detail  that  can  be  used  to  find  specific  features,  and 
therefore  automatically  design  navigation  routes  along  with  tissues  for  surgical  equipment.  This  considerably  reduces 
possible  damage  to  the  patient  [14,  15].  This  strategy  can  also  be  used  to  plan  the  route  of  small  robots  along  the  human 
body  [16]. 

The  strategy  proposed  in  this  article  consists  of  a  general  navigation  scheme  for  small  service  robots  with 
autonomous  movement  capacity  [17,  18].  As  a  robot  platform,  we  use  the  Nao  robot  from  Soft  Bank  Robotics.  The  optical 
sensors  are  the  two  front  cameras  located  in  the  head  of  this  robot.  The  images  are  filtered  for  binarization  and 
morphological  adjustments  to  identify  specific  landmarks  in  the  environment  characterized  by  shape  and  color  [19,  20]. 
From  this  identification,  the  robot  executes  movement  policies  that  allow  it  to  interact  successfully  in  the  environment. 

The  following  part  of  the  paper  is  arranged  in  this  way.  Section  2  presents  preliminary  concepts  and  problem 
formulation.  Section  3  illustrates  the  design  profile  and  development  methodology.  In  Section  4,  we  present  the 
preliminary  results  and  finally  in  Section  5,  we  present  our  conclusions. 
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Let  W  c  R2  be  the  closure  of  a  contractible  open  set  in  the  plane  that  has  a  connected  open  interior  with  obstacles  that 
represent  inaccessible  regions.  Let  Q  be  a  set  of  obstacles,  in  which  each  0  c  O  is  closed  with  a  connected  piecewise- 
analytic  boundary  that  is  finite  in  length.  Furthermore,  the  obstacles  in  Q  are  pair  wise-disjoint  and  countably  finite  in 
number.  Let  E  c  W  be  the  free  space  in  the  environment,  which  is  the  open  subset  of  W  with  the  obstacles  removed. 

We  place  an  agent  (autonomous  robot)  in  the  free  space  of  this  environment.  This  agent  can  know  the 
environment  from  observations  using  its  sensors.  These  observations  allow  it  to  build  an  information  space  /.  An 
information  mapping  is  of  the  form: 

q'-  E  — >  S  (1) 

where, 

S  denotes  an  observation  space,  constructed  from  sensor  readings  over  time,  i.e.,  through  an  observation  history  of 

the  form: 

o:  [0,  t]  — >  S  (2) 

The  interpretation  of  this  information  space,  i.e,  /  X  S  — >  7,  is  that  which  allows  the  agent  to  make  decisions.  The 
agent  performs  readings  of  the  medium  through  its  sensors  forming  a  temporal  sequence  of  values,  which  is  interpreted  to 
perform  actions  according  to  the  desired  movement  policy. 

We  assume  the  agent  is  able  to  sense  the  proximity,  i.e.,  identify  obstacles  in  the  environment  using  minimal 
information.  The  environment  E  is  unknown  to  the  robot.  Furthermore,  the  robot  does  not  even  know  its  own  position  and 
orientation.  Our  goal  is  to  design  the  control  rules  for  the  robot  in  order  to  independently  solve  navigation  tasks  in  a 
dynamic  and  unknown  environment. 

The  system  is  completely  independent,  i.e.,  there  are  no  actions  on  it  produced  by  some  superior  control  unit, 
internal  or  external  to  the  robot.  The  system  must  actively  seek  the  inherent  characteristic  of  the  target  and  monitor.  Trace 
information  is  comprised  of  marks  on  the  navigation  environment,  landmarks,  recognizable  by  its  geometric  shape  and 
color.  This  concept  can  be  extended  to  any  other  recognizable  trace  information. 

According  to  the  local  information  identified  by  the  robot  in  the  environment,  a  control  mode  is  defined  as  the 
determinant  of  the  reactive  behavior  of  the  robot.  A  control  mode  is  a  mapping  u:C  — >  {t,  U]  that  assigns  behavior  to  each 
possible  landmark  identified  in  the  environment.  These  behaviors  are  defined  according  to  the  task  of  the  robot,  for 
example,  moving  towards  the  landmark,  moving  away  from  it  or  dodging  it.  IJ  denotes  the  set  of  all  possible  control  modes 
defined  for  the  robot. 

METHODOLOGY 

Our  recognition  scheme  uses  traditional  strategies  to  identify  shapes  and  colors  in  images  through  digital  image  processing. 
The  overall  operation  is  detailed  in  the  block  diagram  in  figure.  1 . 
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Figure  1:  Functional  Description  of  the  Identification  Algorithm. 

Our  scheme  uses  the  two  cameras  of  the  Nao  robot  (top  and  bottom).  The  code  is  implemented  in  the  Nao  robot 
using  Naoqi.  The  video  from  the  cameras  is  captured  at  15  frames  per  second  in  RGB  color  model  (color  model  in  which 
Red,  Green  and  Blue  light  are  added  together)  with  a  frame  size  of  640*480  pixels  (kVGA  resolution).  The  frames  are  not 
scaled,  all  image  processing  is  done  in  the  same  resolution.  All  images  are  captured  and  stored  in  PNG  (Portable  Network 
Graphics)  format. 

The  first  filter  applied  to  each  frame  is  the  binarization  of  the  image  in  two  colors.  This  binarization  is  done  with 
Open  CV  in  the  HSV  color  space  (Hue,  Saturation,  Value;  alternative  representations  of  the  RGB  color  model)  using  as 
pattern  a  color  involving  yellow,  red,  or  blue.  Then,  we  perform  morphological  image  processing  on  the  images  to  identify 
basic  geometric  shapes.  The  initial  tests  have  been  developed  with  yellow  circles. 

Once  the  regions  of  interest  have  been  identified,  they  are  labeled  and  characterized.  Using  Numpy  matrix 
operations,  the  2D  location  of  the  object  in  the  image  is  identified.  For  verification,  this  information  is  superimposed  on  the 
original  image  captured  by  the  Nao's  camera  (the  region  is  marked  with  a  circle  and  its  center,  from  which  the  three- 
dimensional  coordinates  are  also  defined).  Then,  we  transform  the  2D  location  to  an  absolute  distance  using  the  principle 
of  ranging.  The  estimation  is  not  completely  accurate  due  to  the  lack  of  information  regarding  the  depth;  however, 
combining  the  information  from  the  two  cameras  achieves  a  value  quite  close  to  the  real.  Finally,  the  Nao  robot  is 
programmed  to  respond  in  coherence  with  the  identified  object  (walk  to  the  ball).  This  last  step  consists  of  transferring  the 
estimated  3D  measurements  from  the  images  into  a  3D  location  system  on  the  environment,  which  allows  the  definition  of 
movement  policies  to  the  robot’s  joints. 

All  our  searches  and  recognition  schemes  are  written  in  Python  3.7.3  with  the  use  of  Open  CV  4.1.0.25,  Numpy 
1.16.2,  Pillow  5.4.1  and  Naoqi.  Figure  2  shows  the  result  of  one  of  the  laboratory  tests  (object  to  recognize:  yellow  ball). 
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Figure  2:  Operation  of  the  Algorithm  in  the  Laboratory,  (a)  The  Three  Balls  Used  for 
Evaluation:  Yellow,  Green  and  Red.  (b)  Image  Binarized  by  HSV  Color  Space,  (c) 
Identification  of  Regions  by  Morphological  Adjustment  and  (d)  Initial  Image  with 
Superimposed  Localization  Information. 


RESULTS  AND  DISCUSSIONS 


The  tests  were  performed  on  our  robotic  platform.  Our  assistive  robot  consists  of  two  robotic  platforms:  A  humanoid  Nao 
robot  from  Soft  Bank  Group  for  interaction  with  humans  and  the  environment  and  an  ARMOS  Turtle  Bot  1  robot  from  the 
ARMOS  research  group  for  indoor  navigation  (Figure  3).  Communication  with  the  two  platforms  is  via  a  Wi-Fi 
connection.  The  Nao  robot  is  used  for  image  capture,  real-time  processing  and  as  an  interface  with  the  environment 
(interaction  with  arms  and  head  movement),  while  the  ARMOS  Turtle  Bot  1  robot  is  used  for  platform  navigation  in  the 
environment.  In  our  initial  tests,  the  global  control  of  the  scheme  is  performed  by  the  Nao  robot,  which  informs  the 
ARMOS  Turtle  Bot  1  robot  how  to  move  in  the  environment;  however,  the  future  development  contemplates  performing 
the  processing  and  control  on  the  control  card  of  the  ARMOS  Turtle  Bot  1  robot. 

We  evaluate  the  performance  of  the  strategy  in  the  laboratory  with  different  configurations  varying  the  position  of 
the  balls,  distances  to  the  robot,  number  of  balls  and  even  different  lighting  conditions.  Despite  the  great  possibilities 
offered  by  the  environment,  the  algorithm  was  always  able  to  correctly  identify  the  object  of  interest.  In  some  frames,  the 
algorithm  confused  the  ball  with  the  environment  when  the  light  conditions  were  particularly  poor;  however,  from 
neighboring  images,  it  was  possible  to  establish  the  2D  location  of  the  robot  in  100%  of  cases. 
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of  a  Humanoid  Nao  Robot  from  Soft  Bank  Group  at  the  Top  and  an 
ARM  OS  Turtle  Bot  1  Tank  Robot  from  the  ARM  OS  Research  Group  at  the 

Bottom. 

To  define  the  relative  position  of  the  object  of  interest  or  landmark  to  the  robot,  we  define  a  three-dimensional 
rectangular  coordinate  system  with  origin  in  the  robot  camera  (Figure.  4).  The  T-axis  grows  positively  to  the  right  of  the 
robot,  the  z-axis  grows  positively  to  the  front  of  the  robot,  and  the  v-axis  grows  positively  upwards  from  the  robot  head. 
This  means  that  elements  on  the  ground  will  have  a  negative  component  in  the  v-axis  (they  are  below  the  level  of  the  head. 
Figure.  2d).  The  camera  has  a  viewing  angle  of  70°,  which  together  with  the  size  of  the  robot  defines  a  detection  field  in 
front  of  the  robot  for  the  definition  of  reactive  movements. 


ARMOS  Turtle  not  I 


Figure  4:  Three-Dimensional  Reference  System  with  Origin  in  the  Optical 
Sensor  of  the  Robot  used  to  Establish  the  Relative  Position  of  the  Landmark  to 

the  Robot. 


Figure  2  shows  the  results  for  a  performance  test  under  controlled  conditions.  We  perform  many  of  these  tests  to 
evaluate  the  performance  of  the  scheme.  Table  1  shows  a  summary  of  20  of  these  tests  performed  in  the  same  environment 
under  the  same  lighting  conditions.  In  these  tests,  we  change  the  location  of  the  target  object  to  different  points  in  the 
environment,  all  within  the  robot’s  field  of  vision.  The  code  used  was  always  the  same,  identifying  the  yellow  ball.  The 
identification  of  the  landmark  was  achieved  in  100%  of  the  cases.  Also,  the  estimation  error  to  the  actual  distance  to  the 
landmark  averaged  less  than  3%.  A  slight  increase  in  the  error  is  observed  when  the  distances  increase,  but  the  error  value 
never  exceeds  7%  (Figure  5). 
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Table  1:  Outline  of  20  Laboratory  Tests  in  which  the  Position  of  the  Landmark  is  Estimated  in  Different 

Situations,  all  within  the  Robot’s  Field  of  Vision 


Estimated  three-dimensional  coordinates 

Estimated  straight 

Actual  distance  to 

Estimation 

x[m] 

y[m] 

z[m] 

line  distance  [m] 

sensor [m] 

error  [%] 

0.09 

-0.86 

1.82 

2.01 

1.96 

2.85 

-0.15 

-0.75 

1.51 

1.69 

1.63 

4.09 

-0.36 

-0.79 

1.93 

2.12 

1.98 

6.66 

0.12 

-0.91 

2.03 

2.23 

2.17 

2.82 

0.26 

-0.49 

0.56 

0.79 

0.77 

2.42 

0.39 

-0.56 

0.76 

1.02 

1.00 

1.78 

-0.43 

-0.78 

0.83 

1.22 

1.19 

2.05 

-0.05 

-0.76 

0.79 

1.10 

1.09 

0.24 

-0.18 

-0.81 

0.86 

1.20 

1.15 

4.08 

0.15 

-0.78 

1.33 

1.55 

1.52 

1.89 

0.38 

-0.67 

1.56 

1.74 

1.65 

5.39 

0.49 

-0.69 

1.79 

1.98 

1.93 

2.80 

0.21 

-0.40 

0.69 

0.82 

0.79 

3.55 

0.08 

-0.43 

0,46 

0.63 

0.63 

0.72 

0.26 

-0.35 

0.28 

0.52 

0.52 

0.38 

0.36 

-0.48 

0.81 

1.01 

0.99 

2.31 

0.42 

-0.46 

0.68 

0.92 

0.89 

3.38 

-0.13 

-0.37 

0,73 

0.83 

0.81 

2.02 

-0.26 

-0.45 

0.76 

0.92 

0.89 

3.56 

0.38 

-0.42 

0.97 

1.12 

1.10 

2.46 

Average  error: 

2.77 

Average  error 

0.5  1  1.5  2  2.5 


Direct  distance  from  sensor  to  landmark  [m] 

Figure  5:  Error  Behavior  Concerning  the  Distance  Between  the 
Optical  Sensor  and  the  Landmark. 

The  reactions  programmed  in  the  robot  as  a  response  to  the  identification  of  a  landmark  correspond  to  the  basic 
movements  of  the  robot  in  the  environment  coherent  with  its  task.  In  principle,  we  develop  the  task  of  picking  up  a  ball  and 
carrying  it  to  the  trash  can,  so  the  control  modes  programmed  as  a  response  include:  walking  to  the  landmark,  moving 
away  from  the  landmark  and  dodging  the  landmark.  These  tasks  were  scheduled  parallel  to  our  identification  and  tracking 
scheme  and  presented  an  excellent  performance  in  real-time  (without  delays  that  would  affect  the  task’s  performance). 
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From  the  results,  it  is  proposed  to  improve  the  algorithm  by  including  stereoscopic  vision.  In  our  platform,  we 
have  the  problem  of  incorporating  a  system  of  two  cameras  to  the  robot,  or  in  its  defect,  to  add  some  sensor  that  is  able  to 
inform  about  the  depth  to  the  object  of  interest. 

CONCLUSIONS 

To  develop  a  protocol  for  the  development  of  applications  for  an  assistive  robot,  we  present  in  this  article  a  strategy  for  the 
identification  of  characteristics  in  the  environment,  from  which  it  is  possible  to  define  the  development  of  parallel  tasks 
and  movement  control  policies.  The  algorithm  uses  Open  CV  to  identify  the  elements  of  interest  from  colors  and  shapes. 

In  particular,  we  have  evaluated  the  operation  by  filtering  through  yellow,  blue  and  red  colors,  and  for  circular 
shapes.  The  tests  were  performed  with  balls  of  different  colors  within  reach  of  the  robot’s  cameras.  The  scheme  uses  color 
binarization  and  morphological  adjustment  over  the  regions  to  determine  the  target  point.  Once  the  area  has  been  identified 
in  the  2D  image,  this  information  is  tagged  and  transformed  into  3D  location  to  coordinate  the  robot’s  movement. 

The  performance  tests  were  performed  on  a  Nao  V5  Evolution  robot  equipped  with  an  Intel  Atom  @1.6  GHz 
processor  and  OS  NAOqi  2.0.  The  cameras  on  the  head  of  the  Nao  robot  were  used  as  optical  sensors  in  the  strategy.  The 
code  was  developed  in  Python  and  Open  CV  Laboratory  tests  showed  high  algorithm  performance  and  very  low 
computational  cost.  The  error  in  the  estimation  of  the  distance  from  the  sensor  to  the  target  never  exceeded  7%  with  respect 
to  the  actual  distance.  In  addition,  the  error  is  reduced  as  the  distance  to  the  target  decreases.  Besides,  integration  with 
parallel  tasks  could  be  carried  out  without  causing  operational  delays. 

In  order  to  reduce  the  error  in  the  estimation,  we  propose  to  integrate  in  the  future  to  the  ARMOS  Turtle  Bot  1 
robot  a  scheme  of  two  cameras  with  a  disposition  similar  to  the  one  of  the  eyes  in  people  and  animals,  this  with  the  purpose 
of  adding  information  to  the  scheme  related  to  the  depth  at  which  the  landmark  is  located.  This  goes  hand  in  hand  with  the 
migration  of  the  control  scheme  to  a  higher  capacity  processing  unit  located  in  this  robot. 
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